{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "OlDwW4HY8MoU"
   },
   "source": [
    "# Introduction\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "2pFlZCUv7hM-"
   },
   "source": [
    "\n",
    "In this notebook, we will \n",
    "- Learn how to use BoostedTrees Classifier for training and evaluating\n",
    "- Explore how training can be speeded up for small datasets\n",
    "- Will develop intuition for how some of the hyperparameters affect the performance of boosted trees.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "QGP_iZh-1SX3"
   },
   "outputs": [],
   "source": [
    "# We will use some np and pandas for dealing with input data.\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "# And of course, we need tensorflow.\n",
    "import tensorflow as tf\n",
    "\n",
    "from distutils.version import StrictVersion"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tf.__version__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "vfxkZE-MaY0h"
   },
   "source": [
    "# Load dataset\n",
    "We will be using the titanic dataset, where the goal is to predict passenger survival given characteristiscs such as gender, age, class, etc."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "gd995mWZzOTz"
   },
   "outputs": [],
   "source": [
    "tf.logging.set_verbosity(tf.logging.INFO)\n",
    "tf.set_random_seed(123)\n",
    "\n",
    "# Load dataset.\n",
    "dftrain = pd.read_csv('https://storage.googleapis.com/tf-datasets/titanic/train.csv')\n",
    "dfeval = pd.read_csv('https://storage.googleapis.com/tf-datasets/titanic/eval.csv')\n",
    "y_train = dftrain.pop('survived')\n",
    "y_eval = dfeval.pop('survived')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "HPs8YoHMkB7_"
   },
   "outputs": [],
   "source": [
    "fcol = tf.feature_column\n",
    "CATEGORICAL_COLUMNS = ['sex', 'n_siblings_spouses', 'parch', 'class', 'deck',\n",
    "                       'embark_town', 'alone']\n",
    "NUMERIC_COLUMNS = ['age', 'fare']\n",
    "\n",
    "def one_hot_cat_column(feature_name, vocab):\n",
    "  return fcol.indicator_column(\n",
    "      fcol.categorical_column_with_vocabulary_list(feature_name,\n",
    "                                                 vocab))\n",
    "fc = []\n",
    "for feature_name in CATEGORICAL_COLUMNS:\n",
    "  # Need to one-hot encode categorical features.\n",
    "  vocabulary = dftrain[feature_name].unique()\n",
    "  fc.append(one_hot_cat_column(feature_name, vocabulary))\n",
    "\n",
    "for feature_name in NUMERIC_COLUMNS:\n",
    "  fc.append(fcol.numeric_column(feature_name,\n",
    "                                dtype=tf.float32))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "R_51OufwaY0o"
   },
   "outputs": [],
   "source": [
    "# Prepare the input fn. Use the entire dataset for a batch since this is such a small dataset.\n",
    "def make_input_fn(X, y, n_epochs=None, do_batching=True):\n",
    "  def input_fn():\n",
    "    BATCH_SIZE = len(y)  # Use entire dataset.\n",
    "    dataset = tf.data.Dataset.from_tensor_slices((X.to_dict(orient='list'), y))\n",
    "    # For training, cycle thru dataset as many times as need (n_epochs=None).    \n",
    "    dataset = dataset.repeat(n_epochs)  \n",
    "    if do_batching:\n",
    "      dataset = dataset.batch(BATCH_SIZE)\n",
    "    return dataset\n",
    "  return input_fn"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "DMwL7qlrAdWk"
   },
   "source": [
    "# Training and Evaluating Classifiers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "pBhdfNzXjAsT"
   },
   "outputs": [],
   "source": [
    "TRAIN_SIZE = len(dftrain)\n",
    "params = {\n",
    "  'n_trees':10,\n",
    "  'center_bias':False,\n",
    "  'l2_regularization':1./TRAIN_SIZE # regularization is per instance, so if you are familiar with XGBoost, you need to divide these values by the num of examples per layer\n",
    "}\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Exercise: Train a Boosted Trees model using tf.estimator. What are the best results you can get?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "hw4avA1R23dL"
   },
   "source": [
    "Train and evaluate the model. We will look at accuracy first.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 852
    },
    "colab_type": "code",
    "id": "GsMoeNiEHlox",
    "outputId": "ac640831-a46a-4c6a-b901-6663aa9f2ee9"
   },
   "outputs": [],
   "source": [
    "# Training and evaluation input functions.\n",
    "n_batches_per_layer = 1  # Use one batch, consisting of the entire dataset to build each layer in the tree.\n",
    "DO_BATCHING = True\n",
    "\n",
    "train_input_fn = make_input_fn(dftrain, y_train, n_epochs=None, do_batching=DO_BATCHING)\n",
    "eval_input_fn = make_input_fn(dfeval, y_eval, n_epochs=1, do_batching=DO_BATCHING)\n",
    "est = # TODO\n",
    "\n",
    "est.train(train_input_fn)\n",
    "\n",
    "# Eval.\n",
    "pd.Series(est.evaluate(eval_input_fn))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "eE1xyZrt285E"
   },
   "source": [
    "Exercise #2:  Can you get better performance out of the classifier? How do the results compare to using a DNN? Accuracy and AUC?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "fDNzuC0xUstP"
   },
   "source": [
    "# Results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "d_iUYaTq2ZgL"
   },
   "source": [
    "Let's understand how our model is performing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "height": 289
    },
    "colab_type": "code",
    "id": "kgds_rmq2_2t",
    "outputId": "931df046-e7fe-4e7e-9680-5106a40265ad"
   },
   "outputs": [],
   "source": [
    "pred_dicts = list(est.predict(eval_input_fn))\n",
    "probs = pd.Series([pred['probabilities'][1] for pred in pred_dicts])\n",
    "\n",
    "probs.plot(kind='hist', bins=20, title='predicted probabilities');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "jq-CPquY-bG3"
   },
   "source": [
    "**???** Why are the probabilities right skewed?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "qrvEyh4Q3YgC"
   },
   "source": [
    "Let's plot an ROC curve to understand model performance for various predicition probabilities."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "height": 307
    },
    "colab_type": "code",
    "id": "ByhMg-_a3K_q",
    "outputId": "f02d1b06-cc0a-43ac-b95e-3e9192a59099"
   },
   "outputs": [],
   "source": [
    "from sklearn.metrics import roc_curve\n",
    "from matplotlib import pyplot as plt\n",
    "\n",
    "fpr, tpr, _ = roc_curve(y_eval, probs)\n",
    "plt.plot(fpr, tpr)\n",
    "plt.title('ROC curve')\n",
    "plt.xlabel('false positive rate')\n",
    "plt.ylabel('true positive rate')\n",
    "plt.xlim(0,)\n",
    "plt.ylim(0,);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "9HKAt75V3O8E"
   },
   "source": [
    "**???** What does true positive rate and false positive rate refer to for this dataset?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Copyright 2019 Google Inc. Licensed under the Apache License, Version 2.0 (the \"License\"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "name": "ASL_b_boosted_trees_estimator",
   "provenance": [],
   "version": "0.3.2"
  },
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
